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SUMMARY 


Problem 

Performance  measurement  is  antecedent  and  fundamental  to  the  conduct  of  person¬ 
nel  research  and  personnel  operations  in  military  systems.  It  has  a  significant  role  to  play 
in  regard  to  personnel  acquisition,  training,  duty,  assignment,  classification,  and  separa¬ 
tion.  In  view  of  the  importance  of  performance  measurement  to  the  military  services, 
new  methods  of  measuring  performance  that  may  be  conveniently  and  economically 
utilized  in  operational  situations  must  be  developed. 

Objectives 

The  objectives  of  this  research  were  to  (1)  delineate  the  essential  factors  of 
performance  measurement,  (2)  distinguish  methods  for  developing  performance  measure¬ 
ment  models,  (3)  indicate  how  mathematical  models  for  performance  measurement  are 
developed  from  those  methods,  and  (4)  illustrate  how  these  models  can  be  applied  in 
operational  and  practical  situations  to  obtain  performance  measurements. 

Approach 

To  develop  a  method  to  measure  a  performer's  capability,  the  factors  affecting  the 
measurement  must  be  identified.  The  critical  element  in  the  process  is  the  comparison 
between  the  performer's  capability  and  the  difficulty  of  the  task  to  be  performed.  This 
comparison  and  the  observable  information  obtainable  when  the  performer  actually 
performs  the  operations  of  a  job  are  sufficient  for  defining  mathematical  measurement 
models  of  a  person's  capability  for  several  types  or  mixtures  of  performance  variables. 

Findings 

Three  methods  of  model  development  are  identified  and  two  models  are  specified. 
Model  I  uses  observable  information  expressed  qualitatively  and  Model  II  uses  observable 
information  expressed  quantitatively.  This  latter  model  accommodates  operations  that 
are  associated  with  variables  having  different  distributional  forms,  including  a  mixture  of 
qualitative  and  quantitative  operational  variables.  Examples  are  provided  to  show  how 
each  model  can  be  applied  in  practice. 

Conclusions 


1.  Model  I  is  a  performance  model  that  can  be  efficiently  used  to  measure 
performance  by  means  of  test  items.  It  is  simple  to  understand  and  to  use,  and  has  the 
property  of  specific  objectivity. 

2.  Model  II  is  a  performance  model  that  can  be  effectively  used  to  measure 
performance  of  persons  in  jobs  requiring  a  variety  of  operations,  each  possessing  its  own 
form  of  observational  variable  distribution. 

3.  Job  performance  measurement  models  are  conveniently  developed  as  functions 
of  the  comparison  between  the  performer's  capability  and  the  difficulties  of  the 
operations  to  be  performed  on  the  job. 


Recommendations 

1.  Model  I  should  be  used  to  assess  abilities  of  persons  who  are  to  be  evaluated  by 
tests  in  which  items  are  scored  with  dichotomous  responses. 

2.  Model  II  should  be  used  to  assess  capabilities  of  persons  to  perform  jobs  whose 
operations  are  evaluated  by  means  of  variables  with  different  distributional  forms. 
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INTRODUCTION 


Problem 

Performance  measurement  (PM)  is  viewed  as  fundamental  to  the  conduct  of  personnel 
research  and  operations  in  all  military  systems.  Perhaps  no  organized  group  of  scientists 
appreciates  this  more  than  do  personnel  research  workers.  In  regard  to  personnel,  PM  has 
a  significant  role  to  perform  in  the  following  areas:  (1)  acquisition,  (2)  training,  (3)  duty, 
(4)  assignment,  (5)  classification,  and  (6)  separation. 

Objectives 

The  objectives  of  this  research  were  to  (1)  delineate  the  essential  factors  of 
performance  measurement,  (2)  distinguish  methods  for  developing  performance  measure¬ 
ment  models,  (3)  indicate  how  mathematical  models  for  performance  measurement  are 
developed  from  those  methods,  and  (U)  illustrate  how  these  models  can  be  applied  in 
operational  and  practical  situations  to  obtain  peformance  measurements. 


Background 

Consider  the  logistic  function  of  some  variable  t(ni)  to  be  defined  as 


f  t(ni) 


_ 1 _ 

1  +  exp  [-t(ni)] 


1  +  exp  [t(ni)] 


(1) 


=  exp  [t(ni)] /D(ni) 


From  the  table  below,  it  can  be  noted  that  equation  (1)  "behaves"  like  the  probability 
distribution  function  (also  referred  to  as  the  cumulative  distribution  function  (cdf)  of 
t(ni)).  This  function  expresses  the  probability  that  the  variable  t  may  take  on  a  value  less 
than  or  equal  to  t(ni).  This  may  be  expressed  as 

f[t(ni)]  =  Pr[t<t(ni)]  .  (2) 

Equation  (1)  may  serve  as  a  distribution  function  provided  it  is  nondecreasing  with 
increasing  t(ni)  and  satisfies  f(-°°)  =  0  and  f(+°°)  =  1.  There  are  some  other  technical 
matters  to  be  considered  in  order  to  be  mathematically  precise  (continuity  and 
integrability).  However,  for  the  present  purposes,  these  are  ignored.  Numerical 
evaluation  of  (1)  shows  the  following: 


1 


t(ni) 

exp  ft(ni) 

D(ni) 

Pr  ft(ni) 

-  00 

.00000 

1.00000 

.00000 

-10 

.00004 

1.00004 

.00004 

-5 

.00674 

1.00674 

.00669 

-2 

.13534 

1.13534 

.11920 

-1 

.36788 

1.36788 

.26894 

0 

1.00000 

2.00000 

.50000 

1 

2.71828 

3.71828 

.73106 

2 

7.38906 

8.38906 

.88080 

5 

148.41316 

149.41316 

.99331 

10 

22026.456 

22027.456 

.99995 

+  <o 

OO 

OG 

1.00000 

Equation  (1)  can  be  shown  to  have  other  desirable  properties.  For  example,  if  (2)  is 
abbreviated  as: 


P(ni) 

=  Pr 

[t<t(ni)J 

=  exp  (t(ni)J  /D(ni)  ,  and 

(3) 

Q(ni) 

=  1  - 

•  P(ni) 

(4) 

=  1/D(ni), 

then, 

P(ni) 

exp  {t(ni)l  ,  and 

(5) 

Q(ni) 

the  natural  logarithm 

of  (5)  is 

L(ni) 

=  In 

P(ni)~1  =  t(ni). 

_Q(ni)J 

(6) 

Consider  also  two 

i  different  values  of  i,  say  i  and  j,  i  £  j: 

P(ni)*Q(nj)  = 

exp  [t(ni)j  /D(ni)*D(nj)  ,  and 

(7) 

P(nj)*Q(ni)  = 

exp  |t(nj)3  /D(ni)*D(nj). 

(8) 

Then, 

R(nij)  ''  Pfr$Q(nij  =  exP  &(ni)  -  t(ni)3  * 

Finally,  the  natural  logarithm  of  R(nij)  is 


(9) 


L(nij)  =  t(ni)  -  t(nj). 


(10) 


For  particular  definitions  of  t(ni),  these  and  other  properties  of  (1)  are  very  useful. 


PERFORMANCE  MEASUREMENT 


Measurement  Process 

Earlier,  Panos  (1969)  remarked  that  the  process  of  taking  measurements  is  a  process 
of  making  comparisons.  This  concept  has  been  augmented  by  the  author  with  a  feature 
that  contributes  to  understanding  of  the  topic  of  "measurement."  More  explicit 
information  about  this  feature  will  be  reported  later  in  the  report. 

The  technical  discussion  of  PM  begins  with  two  short  and  innocuous  statements 
related  to  PM: 

1.  Statement  A.  The  evaluator's  goal  in  PM  is  to  measure,  on  a  quantitative  scale, 
the  property  of  a  performer,  called  capability,  that  performers  exhibit  while  interacting 
with  contexts  sampled  from  a  situation  of  interest  to  the  evaluator. 

2.  Statement  B.  A  pciiormer  is  said  to  have  capability  to  the  extent  that  the 
interactions  with  the  contexts  exhibit  information  by  which  the  level  of  capability  may  be 
ascertained. 

From  these  statements,  the  essential  factors  from  which  models  of  PM  may  be 
constructed  will  be  identified. 

Factors  of  PM 


The  Situation 


Statement  A  implies  that  an  evaluator's  purpose  is  to  measure  a  performer's 
capability  with  respect  to  some  situation  of  interest  to  the  evaluator.  A  "situation"  is  a 
set  of  conditions,  activities,  or  circumstances  (contexts)  that  collectively  characterize  an 
important  or  relevant  state  of  affairs.  To  be  more  definite,  important  situations  are 
often  "jobs"  and  the  contexts  of  these  are  the  set  of  operations  that  must  be  undertaken 
to  perform  each  job.  Sometimes  a  training  program  is  a  situation  and  contexts  are 
particular  types  of  learning. 

Performer  Capability 

Both  statements  refer  to  a  property  of  a  performer,  called  his  capability.  We  assume 
that  this  property  controls  how  well  persons  perform  the  jobs  they  undertake.  A  liberal 
view  of  this  concept  would  be  that  it  might  also  be  a  function  of  the  performer’s 
motivation  (or  other  psychological  factors)  that  might  significantly  influence 
performance.  These  other  factors  are  neither  separately  identified  nor  recognized  in  the 
models  developed  in  this  report.  It  is  assumed  that  the  capability  sufficiently  represents 
the  concept. 

The  capability  of  person  (n)  is  such  an  important  component  in  PM  that  it  is  a 
parameter  of  PM  models  and  is  given  the  symbol  a(n).  Its  measurement  is  the  raison 
d'etre  for  the  PM  process. 
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Operation  Difficulty 

The  statements  also  imply  that  the  operations  performed  while  doing  a  job  have  some 
difficulty,  or  else  persons  would  not  require  a  capability  in  order  to  perform  them.  The 
"difficulty"  of  operation  (i)  is  also  a  parameter  and  is  given  the  symbol  6 (i). 

Comparison 

In  the  Measurement  Process  section,  the  concept  of  a  comparison  was  mentioned.  We 
define  the  relevant  comparison  for  the  PM  process  as 

t(ni)  =  a(n)  -  6(i).  (11) 

This  is  the  feature  that  contributes  to  an  understanding  of  the  PM  process.  If  the 
difficulty  of  an  operation  is  high,  the  concept  presented  here  is  that  a(n)  must  be  even 
greater  if  the  performer  is  likely  to  perform  the  operation  effectively.  Performances 
associated  with  negative  values  of  t(ni)  have,  by  (2),  a  relatively  small  probability  of  good 
performance  on  the  given  operation. 

The  Observables 

Statement  B  indicates  that  a  person's  performance  with  respect  to  an  operation 
results  in  some  information,  usually  behavioral  evidence,  which  indicates,  in  some  sense, 
the  capability  of  the  performer  with  respect  to  the  difficulty  of  the  operation.  When 
appropriate,  the  characterization  of  this  information  is  symbolized  for  person  (n)  and 
operation  (i)  as  X(ni)  or  X(i). 

It  is  the  contention  of  this  report  that  the  three  factors,  ot(n),  <5(0,  and  X(ni),  are  the 
essential  factors  for  developing  models  for  PM.  The  factor  a(n)  specifies  a  latent 
(unobservable)  trait  of  a  performer,  6(i)  characterizes  a  latent  trait  of  the  operation 
performed,  and  X(ni)  provides  an  observable  link  that  may  mathematically  relate  the  first 
two  factors  and,  hence,  provide  a  means  by  which  each  may  be  estimated  and  interpreted. 
Any  capability  afforded  by  PM  to  estimate  a(n)  quantitatively  is  extremely  significant  for 
personnel  research  as  it  immediately  provides  measurement  for  which  the  powerful  tools 
of  quantitative  statistical  methods  may  be  employed  for  solving  problems  associated  with 
the  aspects  of  personnnel  research  and  operations  mentioned  in  the  Introduction. 

The  observable  performance  information  may  be  characterized  in  many  different 
ways.  Five  of  these  ways  are  listed  below. 

1.  Performance  Quality.  The  outcome  of  a  performance  may  be  considered  from 
the  point  of  view  of  its  quality.  The  quality  may  be  thought  of  as  being  established  on  the 
basis  of  judgment  or  by  more  objective  evidence.  In  any  event,  the  quality  of 
performance  may  be  indicated  by  classification.  The  performance  may  be  classified  in 
one  of  K  categories  of  quality.  One  of  the  PM  models  reported  here  uses  exactly  this  type 
of  observation. 

2.  Performance  Quantity.  Similarly,  the  fortunate  evaluator  may  acquire  quantita¬ 
tive  information.  Such  may  also  be  utilized  in  a  PM  model.  In  this  case,  the  measurement 
problem  is  primarily  a  statistical  problem  of  ascertaining  the  statistical  distribution  of 
the  X(ni).  Likely  candidates  are  the  normal,  beta,  and  gamma  distributions  as  variations 
in  their  parameters  provide  the  great  variety  for  forms  observed  in  practice. 
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3.  Performance  Counts.  Still  another  type  of  variable  that  might  represent 
performance  information  are  variables  resulting  from  counting  operations.  Such  counts 
may  either  represent  performance  errors  or  attainments  or  both.  Such  counting  data  may 
have  Poisson  distributions. 

4.  Performance  Time.  Performance  time  is  an  obvious  choice  for  characterizing 
performance  information.  Its  distribution  is  difficult  to  specify  in  general,  but  it  seems 
likely  that  one  of  the  distributions  mentioned  previously  in  these  descriptions  may  be 

appropriate. 

5.  Mixed  Performance  Variables.  Some  situations  may  allow  each  operation's 
performance  to  be  represented  by  performance  variables  of  the  same  type,  but,  in 
general,  this  seems  unlikely.  Usually,  a  "job"  requires  that  several  operations  be 
performed  and  each  may  be  characterized  by  a  different  type  of  performance  variable. 
The  method  described  in  this  report  of  measuring  a(n)  must  and  does  allow  for  this 
circumstance  to  occur.  This  facility  is  permitted  because  each  performance  variable  is 
transformed  to  a  common  metric — its  cumulative  probability  that  the  mathematical 
model  of  the  next  section  relates  to  the  PM  parameters  a(n)  and  6 (i)  independently  of  the 
type  of  performance  variable. 

PM  Models 

Three  categories  of  methods  for  developing  measurement  models  are  presented. 

Method  1 

This  form  was  originally  defined  and  used  by  Rasch  (1960)  to  develop  a  measurement 
model  for  measuring  capability  by  observing  dichotomous  responses  made  by  performers 
of  operations  (achievement  test  items).  The  feature  of  note  here  is  that  the  Method  I 
type  of  PM  model  incorporates  the  comparison  (11)  and  the  observable  X(ni)  into  the  same 
expression.  This,  in  turn,  endows  the  model  with  a  property  called  "objectivity,"  meaning 
that  the  a(n)  estimates  are  independent  of  which  operations  are  used  and  that  6(i) 
estimates  are  independent  of  which  performers  are  involved  (see  Wright,  1967  and 
Choppin,  1968). 

Method  II 

The  form  of  this  type  of  PM  model  is  introduced  in  this  report.  Its  distinctive  aspect 
is  that  it  is  recognized  that  the  function  (1)  of  the  comparison  (11)  is  a  cumulative 
probability  and  this  probability  is  set  equal  to  the  percentile  rank  (another  name  for 
cumulative  probability)  of  an  observed  performance  variable.  The  unique  idea  is  that, 
instead  of  assuming  that  a  function  of  observed  performance  variables  is  equal  to 
capability,  the  model  assumes  that  that  observable  function  equals  a  function  of  the 
comparison  (11).  In  effect,  the  model  says  that  the  performance  observed  depends  upon 
the  difference  between  the  capability  of  the  performer  and  the  difficulty  of  the  operation 
being  performed  and  not  only  upon  the  performer's  capability. 

Method  HI 

This  form,  introduced  by  Wright  and  Mead  (1978),  substitutes  (1)  for  the  probability  in 
the  distribution  functions  of  the  performance  variable  that  is  observed.  The  authors  do 
this  for  a  variable  with  a  binomial  distribution  in  their  report. 


RESULTS  AND  DISCUSSION 
Comparison,  Capability,  and  Difficulty  Estimation 

For  purposes  of  continuity,  an  application  of  Model  ll  is  provided  before  addressing 
the  application  of  Model  I.  Next,  we  will  show  how  to  estimate  (11)  for  a  Method  II  model 
and  the  two  parameters  comprising  (11).  First,  we  have  by  (6)  that 

L(ni)  =  a(n)  -  6(i).  (12) 

Equation  (12)  implies  that  a(n)  does  not  depend  upon  (i).  This,  in  turn,  implies  that 
Model  II  assumes  the  existence  of  a  single  job  performance  capability  and  not  a 
multidimensional  set  of  abilities- -one  for  each  job  operation.  Those  separate  abilities 
may  be  estimated  by  applying  Model  II  to  each  job  operation  separately.  In  general,  if  a 
job  consists  of  I  operations,  (12)  can  be  summed  over  these  operations  yielding  from  (12): 


I  I 

£  L(ni)  =  Ia(n)  -  Z  6(i).  (13) 

i=l  i  =  l 

Therefore,  we  can  estimate  a(n)  for  a  performer  from 
-  I 

a(n)  =  E  L(ni)  +  6(i)  /I  (14) 

i=l 

where  the  person  performs  I  operations  of  known  difficulty  and  L(ni)  is  estimated  from  (6). 
To  use  (14),  we  need  to  show  how  to  estimate  the  6 (i).  Equation  (14)  simplifies 
considerably  if  we  can  assume  (19)  and  that  the  "job  ability"  is  the  same  for  all  job 
operations. 

Consider,  for  simplicity,  that  a  "job"  is  comprised  of  1=3  operations,  the  performance 
variables  being,  perhaps,  of  1=3  different  types.  We  then  observe  X(ni)  on  each  operation 
for  the  N  members  of  a  calibration  sample  and  array  this  data  as  in  (15). 

Performance  Variables  of  the  Calibration  Sample 

n  X(nl)  X(n2)  X(n3) 


1 

2 

• 


(15) 


N 

From  the  distributions  in  the  columns  of  (15),  the  P(ni)  of  (3)  can  be  found  and 
arranged  as  (16). 


6 


Percentile  of  the  Performance  Variables 


n  P(nl)  P(n2)  P(n3) 


1 

2 

• 

N 


(16) 


For  each  member  of  the  calibration  sample,  the  L(nij)  term  of  (10)  can  be  found  and 
T(ij),  corresponding  to  (18),  can  be  found  as  (17). 

N 

T(ij)  =  EL(nij)  (17) 

n 


1 

2 

I  =3 

1 

— 

EL(nl2) 

n 

EL(nl3) 

n 

CM 

II 

ZL(n21) 

n 

-- 

EL(n23) 

n 

1=3 

EL(n31) 

n 

EL(n32) 

n 

— 

The  T(ij)  column  sums  are  estimates  of  NI6(i)  if  we  assume  (19) 


(18) 


I  a 

E  6(i)  =  0.  (19) 


This  allows  us  to  estimate  6(i)  by  the  sum  of  the  T(ij)  columns  as 


6(0=  E  EL(nij)/I*N,  for  each  j  and  i  £j.  (20) 

Uj  n 

Therefore,  by  (12)  we  can  estimate  the  comparison  (11)  and  by  (14)  we  can  estimate 
capability,  given  the  difficulties  that  can  be  estimated  from  (20).  Note  that  performers 
need  not  perform  each  of  the  I  operations  for  their  capabilities  to  be  evaluated. 

The  above  estimates  are  predicated  on  our  ability  to  estimate  the  P(ni)  and  Q(ni)  of 
(3)  and  (4).  This  may  be  accomplished  by  using  (15)  and  standard  statistical  methods  of 
fitting  data  to  particular  statistical  distributions,  remembering  to  include  a  test  for 
goodness  of  fit  of  the  observed  data  to  the  theoretical  distribution. 
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Example  oi  Application  of  PM  by  Method  II 


As  an  example  of  Model  II  analysis,  consider  PM  for  the  "job"  called  Morse  Code 
operation.  To  make  the  analysis  simple,  we  assume  that  the  calibration  and  evaluation 
samples  consist  of  the  same  N  =  5  persons.  Each  of  these  persons  has  provided  I  =  3 
observations  of  their  performance.  These  are: 

X(ni)  =  300  minus  the  time  in  seconds  required  by  operator  (n)  to  send  a  given 
message. 

X(n2)  =  the  number  of  words  sent/20  by  operator  (n)  in  1  minute.  (21) 

X(n3)  =  S  minus  the  number  of  errors  made  by  operator  (n)  during  his  reception 
of  a  given  message. 

The  evaluator  has  determined  from  analysis  of  a  large  set  of  records  that  the  X(ni) 
observations  have  the  following  distributions: 

X(nl)  are  normally  distributed  with  p  =  101.84  ando  =  19.71. 

X(n2)  have  a  beta  distribution  with  parameters  p=5  and  q=3.  (22) 

X(n3)  have  a  Poisson  distribution  with  parameter  m=2.0. 

The  observed  X(ni)  data  are  given  below  for  N  =  5,  together  with  their  percentiles 
P(ni).  For  this  analysis,  it  is  important  to  scale  the  X(ni)  so  that  increasing  (decreasing) 
magnitude  on  each  variable  signifies  increasing  (decreasing)  capability  on  the  job. 


n 

X(nl) 

P(nl) 

X(n2) 

P(n2) 

X(n3) 

P(n3) 

1 

76 

.050 

.31 

.033 

2 

.677 

2 

152 

.999 

.70 

.647 

8 

.9999 

3 

103 

.529 

.64 

.509 

5 

.983 

4 

147 

.998 

.83 

.899 

7 

.999 

5 

66 

.011 

.22 

.007 

3 

.857 

Having  determined  the  P(ni)  values  from  knowledge  of  the  X(ni)  values  and  their 
distributions,  we  use  (9)  to  compute  (10)  and  array  the  L(nij)  as  T(ij)  as  follows: 


l 

EL(nl2) 

EL(nl3) 

n 

n 

T(ij)= 

2 

EL(n21)  - 

EL(n23) 

n 

n 

(24) 


3 


EL(31)  El(n32) 


For  the  preceeding  data: 


ZL(nl2)  =  -IL(n21)  =  11.297 
n  n 

EL(nl3)  =  -EL(n31)  =  -16.918 
n  n 

ZL(n23)  =  -ZL(n32)  =  -28.210 
n  n 

These  results  give  T(ij)  as: 


T(ij)  = 


-  11.297  -16.918 

-11.297  -  -28.210 

16.918  28.210 


Adding  the  columns  of  T(ij),  we  get  the  vector  T(i)  as: 


T(i)  =  5.621,  39,507,  -45.128  , 


and,  dividing  T(i)  by  N1  =  15,  we  get  the  estimated  operation  difficulties: 


(25) 


(26) 


(27) 


d(i)  =  5(0  =  0.374,  2.634,  -3.008  .  (28) 

Thus,  relatively,  operation  (3)  is  easy,  operation  (2)  is  hard,  and  operation  (1)  is  of 
intermediate  difficulty.  Note  also  that 

Ed(i)  =  0,  (29) 

satisfying  condition  (19)  for  this  job. 

Our  final  arithmetical  task  is  to  estimate  the  a(n)  for  these  operators.  This  is  simply 
done,  for,  by  (6)  and  (11): 


Uni)  =  In 


f  P(ni)"t 

IH 


=  t(ni)  =  <x(n)  -  5(0, 


2L(ni)  =  Ia(n)  -  E6(i), 
i  n 


but  by  (19)  we  estimate  a(n)  as  a(n)  with 
a(n)  =  £3.(ni|J  /I. 


(30) 

(31) 
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The  L(ni)  and  a(n)  for  this  example  are: 


n 

L(nl) 

L(n2) 

L(n3) 

ZL(ni) 

a(n) 

1 

-2.944 

-3.378 

0.740 

-5.582 

-1.861 

2 

6.907 

0.606 

9.210 

16.613 

5.574 

3 

0.116 

0.036 

4.057 

4.209 

1.403 

4 

6.213 

2.186 

6.907 

15.306 

5.102 

5 

-4.499 

-4.995 

1.791 

-7.663 

-2.554 

We  note  that  operators  (2)  and  (4)  have  the  most  capability  for  the  Morse  Code 
operator  job,  whereas  operators  (1)  and  (5)  have  the  least.  Operator  (3)  is  intermediate  to 
these  two  sets  as  measured  by  our  PM  Model  II. 

Examples  of  Application  of  PM  by  Method  I 

The  following  illustration  is  an  application  of  PM  Model  I  to  intelligence  testing. 
Although  this  application  relates  to  a  classical  psychometric  situation  and  problem,  the 
model  applies  equally  well  to  situations  involving  the  measurement  of  performance 
operations.  The  common  concept  of  these  two  situations  is  that  operation  performance 
quality  is  the  observational  variable.  Quality  is  classified  in  one  of  K  categories  for  each 
operation  of  a  "job."  This  may  be  accomplished  either  subjectively  by  the  evaluator  or  his 
agent,  by  objective  evidence,  or  both.  The  quality  categories  are  the  same  for  each 
operation.  The  analytic  details  of  this  PM  model  are  given  in  Moonan  (1979).  For 
simplicity,  a  sample  case  where  K=3  (i.e.,  we  have  K=3  performance  quality  categories 
that  we  label  "outstanding,"  "adequate,"  and  "poor")  was  selected.  The  standards  for 
responses  of  these  qualities  are  given  objective  definitions.  The  1=3  items  for  the 
miniature  intelligence  test  used  in  this  research  have  been  selected  from  various  technical 
sources.  The  item  responses  are  abbreviated  briefly  as  (+  0  -).  The  items  were  selected 
so  that  the  "test"  that  they  comprise  measures  the  latent  trait  called  "reasoning  ability." 
The  first  item  is  a  puzzle  presented  by  Fisher  (1937,  p.  86).  The  puzzle's  solution  requires 
the  construction  of  4x4  Latin  Square  experimental  designs.  The  second  item  is  a  chess 
problem  from  Sweeney  and  Barclay  (1972,  p.  1).  The  chess  problem  requires  the  player 
(White)  to  mate  the  Black  King  in  two  moves.  The  third  item  is  a  problem  in  elementary 
probability  theory  given  by  Rubenstein  (1975,  p.  185).  The  solution  requires,  in  effect,  the 
application  of  Bayes'  theorem.  The  solutions  to  these  items  are  shown  below: 

1.  The  Welsh  lawyer  is  a  45-year-old  married  socialist. 

2.  Q-h3,  K-e4  forced;  R-c4,  MATE.  (34) 

3.  Pr  [Dem  |  Vote  Rep]  =.25. 

Suppose  that  these  three  items  have  each  been  attempted  by  N  =  300  members  of  a 
calibration  sample  and  the  frequencies  F  [i,j,4,m  ],  by  which  the  sample  members  gave 
responses  of  quality  l  to  item  i  and  responses  of  quality  m  to  item  j,  are  portrayed  as 
shown  below: 
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i 

t\m 

Hi 

0 

+ 

- 

if? 

0 

+ 

- 

H3 

0 

* 

m 

136 
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- 

22 

40 

74 

2 

16 

118 

1 
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96 

- 

16 

28 

52 

2 

11 

83 

■ 

- 

- 

68 

11 

20 

37 

1 

8 

59 

22 

16 

11 

49 

- 

- 

1 

6 

42 

2 

40 

28 

20 

- 

88 

- 

1 

10 

77 

74 

52 

37 

- 

- 

163 

3 

19 

141 

■ 

2 

2 

1 

1 

1 

3 

5 

- 

- 

3 

16 

11 

8 

6 

10 

19 

- 

35 

- 

B 

118 

83 

59 

42 

77 

141 

- 

- 

260 

K  3 

We  perform  our  analysis  for  C2  =  K(K-l)/2  =  C2  =  3  pairs  of  response  categories  (i=l, 
m=2),  (£=1,  m=3),  and  (A=2,  m=3).  For  the  first  pair,  we  extract  (36)  from  (35),  giving: 


_ _ 1 

1 

- 

40 

16 

F  [if  j,  1,2  ]  = 

2 

16 

- 

6 

(36) 

3 

2 

1 

- 

From  (35),  we  construct  the  ratios: 

R  [i, j,  1 ,2 ]  =  F  B,j,l,2]/F[i,j,l,2]  as  (38). 

(37) 

1 

2 

3 

1 

- 

2.500 

8.000 

R  [i,j,l,2 ]  =  2 

.400 

- 

6.000 

(38) 

3 

.125 

.167 

- 

We  also  obtain,  in  (40),  the  natural  logarithms  of  the  R(i,j,l,2);  thus, 

InR  (i,j,  1 ,2 ]  =  L [i,j,l,2  ]  (39) 


(35) 


11 


Lli.j.1,21 


-0.916 
-2.079 


2 


0.916 

-1.792 


2.079 

1.792 


Adding  each  column  of  L  [i,j,l,2] ,  we  get  T(i)  as: 

T(i)  =  [-2.995  -.876  3.871].  (41) 

Finally,  dividing  T(i)  by  -3/2  of  (42),  we  get  estimates  d(i)  of  the  operation  difficulties, 
6  (i)  of  (43), 

A(4,n)  =  -I  Q(3)  -  0(2)  ,  =  -3(1  -  1/2)  =  -3/2,  (42) 

where,  for  this  example, 

I  =  3 
0(3)  =  1 

0(2) =  1/2  (43) 

0(1)  =  0 

d(i)  =  [+2.000  +.584  -2.581  ]. 

The  0  values  are  scale  points  on  the  response  distribution.  This  process  is  repeated  for 
response  pairs  (4=1,  m=3)  and  (4=2,  m=3).  This  effort  yields  the  three  sets  of  6 (i) 
estimates  that  we  average  to  get  the  final  d(i)s. 


000 

.58 

226 

.24 

197 

.29 

The  capability  estimates  for  members  of  an  evaluation  sample  are  found  by  a  process 
known  as  expected  value  estimation  (EVE),  as.shown  in  Moonan  (1979).  Moonan  also  found 
that  Model  I  has  the  inrfportant  property  of  "Specific  objectivity."  *  *  '  * 

CONCLUSIONS 

1.  Model  I  is  a  performance  model  that  can  be  efficiently  used  to  measure 
performance  by  means  of  test  items.  It  is  simple  to  understand  and  to  use.  Also,  the 
model  has  the  property  of  specific  objectivity. 

7.  Model  II  is  a  performance  model  that  can  be  effectively  used  to  measure 
•r  m-mance  of  persons  in  jobs  requiring  a  variety  of  operations,  each  possessing  its  own 
form  of  observational  variable  distribution. 


3.  Job  performance  measurement  models  are  conveniently  developed  as  functions 
of  the  comparison  between  the  performer's  capability  and  the  difficulties  of  the 
operations  to  be  performed  on  the  job. 

RECOMMENDATIONS 

1.  Model  1  should  be  used  to  assess  abilities  of  persons  who  are  to  be  evaluated  by 
tests  in  which  items  are  scored  with  dichotomous  responses. 

2.  Model  II  should  be  used  to  assess  capabilities  of  persons  to  perform  jobs  whose 
operations  are  evaluated  by  means  of  variables  with  different  distributional  forms. 
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